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Human machine interaction (HMI) allows persons to control and interact 
with devices. Starting from elementary apparatus which acquires input bio- 
signals to controlling various applications. Medical applications are amongst 
the very important applications of HMI. One of these medical applications is 
assisting fully/partially paralyzed patients to restore movements or freely 
move using exoskeletons or motorized wheelchairs. Helping patients with 
spinal cord injury or serious neurological diseases to restore their 
movements is a key role objective for most researchers in this field. In this 
paper, an EEG-based HMI system is proposed to assist patients with 
tetraplegia/quadriplegia to mentally control a motorized wheelchair so they 
can move freely and independently. EEG power spectrum (a, B, 5, 8, and y) 
from the frontal lobe of brain is recorded, filtered and wirelessly sent to the 
wheelchair to control directions and engine status. Four different 
experiments were conducted using the proposed system in order to validate 
the performance. Two different GUIs scenarios (cross-shaped and horizontal 
bar) were used with the experiments. Results showed that the horizontal bar 
scenario considered more user friendly while the cross-shaped is the more 
suitable for navigation. The implemented system can be equipped with 
modules and sensors such as GPS, ultrasound and accelerometer that 
improve the system performance and reliability. 


This is an open access article under the CC _BY-SA license. 


Corresponding Author: 
Mohammed G. Ayoub 


Department of Computer Technology Engineering, Technical Engineering College 


Northern Technical University 


Mosul, Iraq 


Email: mohammed.ghanim @ntu.edu.iq 


1. INTRODUCTION 


Technology advancement could play a key role for helping people in their daily life especially for 
those who are suffering from movement disorders, which occurs either genetically or through accidents. Such 
disorders cause partial or complete paralysis of the human body [1], [2]. For example, spinal cord injury 
prevents the communication from the brain to all areas of the human body. In such medical condition, the 
brain is correctly functional but the nerves responsible of movements are non-functioning [2], [3]. 
Biomedical signals such as electroencephalogram (EEG) and electromyography (EMG) have been used 
widely nowadays for human machine interaction (HMI), human computer interaction (HCI), and smart 
rehabilitation system [1], [4]-[7]. EEG is a non-invasive approach that is used to record brain activities as 
electrical signals using multiple electrodes on the scalp [8]. These signals can be used in various themes such 
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as keyboard and mouse control, robot arm control and wheelchairs. Artificial intelligent (AI) has also been 
used for developing such systems such as convolutional neural network (CNN), long short-term memory 
(LSTM) and support vector machines (SVMs) [9]-[12]. Researchers have utilized AI in developing smart 
wheelchair systems such as voice-controlled wheelchairs, gesture-controlled wheelchair and eye and voice- 
controlled system. According to the World Health Organization (WHO), there are seventy-five million 
disabled people need wheelchairs on their daily basis, which represents around 1% of the world’s population 
[13]. Therefore, developing such systems could be the appropriate way to serve the life of those people. 

In recent years, there has been considerable interest in developing various kinds of smart 
wheelchairs using recent technologies such as cameras, sensors and wearable EEG headbands. These 
wheelchairs serve disabled people on their daily basis in different situations [14]-[16]. However, some points 
should be considered during developing smart wheelchairs in order to make them appropriate with the need 
of the wide range of disabled people. For example, some studies have tended to design wheelchairs that focus 
on the movement rather than the interaction with their environment. Different techniques have been 
investigated in developing and implementation such systems that serve disabled people. However, few 
researchers have addressed the issue of the cost while developing and implementing wheelchairs that based 
on EEG signal because of the high cost of the medical equipment. Numerous studies used joystick controllers 
to drive the wheelchairs easily such as visual joystick and electric joystick to assist people with different 
disabilities [17]-[19]. However, such technologies are not suitable for people with complete paralysis such as 
quadriplegia condition, which affects all four limbs. 

In [9], [16] developed a voice recognition-based robotic wheelchair using CNNs. Regarding this 
method, it has some limitations for example; background noise affects the user commands. Moreover, this 
kind of systems can be only functioned with partial or incomplete paralyzed person. According to 
Tamplin et al. [20] quadriplegic patients have impaired vocal ability in comparison with the able-bodied. 
Therefore, these methods should be replaced with advanced technologies that can be used with completely 
paralyzed people. This can be solved in a variety of ways, for example using EEG or EOG headbands which 
can be worked with the aforementioned condition easily [21]-[23]. 

Eye-blink detection is an approach that is used in wide range of applications such as human 
computer interaction, alternative communication systems (ACS) and wheelchairs controlling. This approach 
uses different components for example, webcam and biomedical signals including EEG and EOG. 
Chaudhary et al. [24] proposed an eye-blink based predictive text system used CNN and SVM for eye-blink 
detection in real-time using webcam. Li ef al. [25] have also used such approach for controlling wheelchair. 
They acquired eye-blink from a single-channel EOG signal that is used as an on/off switch commands. 

The main goal of this study is to develop an easy-to-use HMI system that is capable to assist fully 
paralyzed people in their daily life. The system is based on a low-cost single channel EEG headband and 
could be used in motorized wheelchair easily. The paper is organized as follows: section 2 illustrates the 
hardware and software of the developed HMI system, which used in this study. Section 3 addresses the 
experimental procedure as well as the obtained results. Section 4 discusses the results of the study while 
section 5 concludes this study. Finally, section 6 shows future work. 


2. METHOD 

In this part, the permanently components, programming and designing process of the HMI proposed 
system Figure 1 are presented in details. The developed EEG system can be used in multiple scenarios 
regarding the time. Two different layouts have been developed for the proposed system graphical user 
interface (GUI) (cross-shaped and horizontal bar); these will be discussed in details later. 


2.1. Hardware structure 

The key motivation of this system is to help fully/partially paralyzed people to move by using their 
brainwaves (attention and meditation levels). This is done by acquiring the subjects’ EEG signal using 
MindWave NeuroSky device to control the wheelchair navigation through a GUI Figures 2(a) and (b). 
Attention and meditation levels of the brain are difficulty controlled to set at specific level, thus, eye-blinking 
is used in this system as a metric to control directions and ignitions of the motorized wheelchair. The 
proposed system is composed of MindWave NeuroSky, MindWave RF dongle, Arduino Mega, TFT LCD 
adapter shield, TFT 320QDT_9341 and motorized wheelchair. MindWave NeuroSky is a device consists of 
eight parts; ear-clip, power switch, battery, ear-arm, flexible headband, sensor tip, sensor arm and ThinkGear 
chip. The principle of work of this device is based on two sensors (forehead sensor and the ear-clip ground 
reference) that detect and filter EEG power spectrum (a, B, 5, 8, and y) from the frontal lobe of brain [23], 
[26]. These values (i.e., a, B, 6, 8, and y) are analyzed through the ThinkGear chip to produce 
attention/meditation and blink levels. Then, these levels are wirelessly sent to the microcontroller board 
(Arduino Mega) as commands in short range via Bluetooth using MindWave radio frequency (RF) dongle. 
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Thin-film-transistor-liquid-crystal-display (TFT LCD) adapter is attached to Arduino Mega board, so the 
TFT 320QDT_9341 display can easily be installed to the microcontroller board. Finally, the system could be 
wired easily to the motorized wheelchair controls so the subject can use and control it by his/her own 
brainwaves. Table | represents the approximate price of the used hardware in the proposed system that can 
be used with disabled people for different applications including wheelchairs. 


MindWave EEG Headset 


Figure 1. The structure of the entire proposed medication dispenser system 


Blink/double blink indicator 


EEG electrode position of MindWave Blink amplitude 


g | 
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Direction buttons Direction caption 
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Figure 2. Placement of EEG (a) headband electrodes and (b) the developed GUI 


Table 1. The approximate price for the proposed system 


S/N Components Description Quantity Approx. price * ($) 
1 Arduino Mega 2560 Processing unit 1 15.75 
2 Ultra HD Colored TFT LCD Screen __ Display unit 1 17.64 
3 MEGA Shield Expansion Control unit 1 4.97 
4 Bluetooth Transceiver HC-05 Control unit 1 6.68 
5 MindWave Mobile EEG headset Control unit 1 109.99 
Total price 155.03 


*Prices are based on local electronics stores 


2.2. GUI design 

The smart wheelchair has a software part as well as the hardware in which the system GUI is 
represented. The aim of designing this GUI is to allow the user to interact with the system easily and 
comfortably. The average values of blinks have been acquired from the EEG headset raw data, which is 
sampled at the rate of 512 Hz for the duration of one second. The blink value depends on a specified 
threshold in the developed algorithm, which is more than the normal blink of human in order to distinguish 
between the user movement/stop commands and the ordinary eye blinks. The value will appear on the top of 
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the GUI with a rectangular indicator that display a single blink or double blinks. A single blink represents 
movement/stop commands while the double blinks represent a stop without the need of choosing it from the 
GUI, which required more time for selection. The double blinks can be used for emergency stopping which is 
requires less time. The selected command will appear in a text box on the GUI, so the user can change it 
quickly in case of missing out the selection. 


3. EXPERIMENTS AND RESULTS 

A pilot study with five subjects was carried out to validate the proposed system and good results are 
achieved. The study includes four different eye-blinking based experiments with set of trials to evaluate the 
system performance. Five able-bodied subjects with the following rates agreed to participate in these 
experiments; (age: 34+4 years; height: 178+7.9 cm; body mass: 72.7+5.7 kg). The next section presents these 
experiments and their results. Figure 3 shows the proposed and implemented GUI design with two different 
scenarios (Figures 3(a) and (b)) in order to figure out the right one for the final design. 


signal formation and rection information and direction Signal information 


MaOORA) 


Proposed scenario (B) 


Implemented scenario (A) Implemented scenario (B) 


Figure 3. The GUI with two different scenarios (a) cross-shaped and (b) horizontal-bar 


3.1. Experiment-1 

In this experiment, the first designing scenario shown in Figure 3(a) was applied to the display. This 
scenario has five buttons (cross-shaped) referring to forward, backward, left, right and stop commands. In 
each trial, the subject was asked to select a specific button on the screen using eye-blinking. Four trials 
(30 seconds each) were conducted to evaluate the system scanning speed and its capability to offer selections 
in specific time. Table 2 shows the capability of each subject to select a specific direction in each trial. 


Table 2. No. of button selections per trial that a subject could select using eye-blinking in the 1* scenario 
(Trials) No. of selections / 30 sec 


Subject No. Trial 1 Trial 2 Trial 3 Trial 4 Pesan 
forward backward left right 
S1 5 3 5 4 0.829 
$2 5 4 4 4 0.433 
$3 3 4 5 3 0.829 
S4 3 4 5 4 0.707 
S5 5 4 4 4 0.433 
Average No. of selections 4.2 3.8 4.6 3.8 0.646 
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3.2. Experiment-2 

In this experiment, the second designing scenario which shown in Figure 3(b) was applied to the 
display. This scenario has five buttons (horizontal bar) referring to forward, backward, left, right, and stop. 
The experimental protocol is as same as experiment-1; in each trial, the subject was asked to select a specific 
button on the screen using eye-blinking. Table 3 shows the capability of each subject to select a specific 
direction in each trial. 


Table 3. No. of button selections per trial that a subject could select using eye-blinking in the 2™ scenario 
(Trials) no. of selections/30 sec 


Subject no. Trial 1 Trial 2 Trial 3 Trial 4 aint 
forward backward left right 

S1 4 5 4 4 0.433 

$2 4 4 4 4 0 

S3 3 4 4 3 05 

S4 4 ) 4 3 0.632 

S5 4 3 3 4 0:5 
Average No. of selections 3.8 4.2 3.8 3.6 0.415 


3.3. Experiment-3 

In this experiment, the first designing scenario shown in Figure 3(a) was applied to the display again. 
The experimental protocol includes four trials in which the subject was asked to follow a specific path 
(directions) on the ground using a motorized wheelchair (its speed 1 m/s). The display with scenario 1 is 
attached to this wheelchair. The path shown in Figure 4 is the one that subjects should follow. The length of 
the path is 21 meters, in which the subject expected to select button F=six times, L=twice, S=zero times, 
R=three times, B=zero times and double blink (D.B)=six times to complete the path. The ideal consumed time 
(when no selection mistakes are committed) for this scenario (cross-shaped) equal to 60 seconds. Subjects may 
commit to types of mistakes when using this system; missing specific selection and select wrong direction. 
Every missed selection will add five seconds (1-cycle scanning time) to the total consumed time. Whilst 
selecting a wrong button (different direction) will require additional time for multi selections until the direction 
is corrected. Table 4 shows the time consumed by each subject to complete the path in four trials. 
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Figure 4. The evaluation path with four different directions 


Ending Point 
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The period is theoretically calculated according to (1) which is based on the system scanning and 
selecting capabilities. 


Consumed Time = 2F + 3L+S+5R+6B+D.B+ X (1) 


Where: 
F=the no. of forward button selections 
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L=the no. of left button selections 

S=the no. of stop button selections 

R=the no. of right button selections 

B=the no. of backward button selections. 
D.B=the no. of double blinking (emergency stop) 
X=the length of the path 


Table 4. the subject consumed timed per trial to complete the path using eye-blinking in the 1* scenario 


' (Trials) consumed time in sec /path Average consumed 
Subject No. . : ‘ : P 
Trial 1 Trial 2 Trial 3 Trial 4 time (sec) 

SI 80 90 85 715 82.5 

82 85 80 80 715 80 

83 85 90 90 80 86.25 

S4 80 80 70 710 715 

S5 715 70 715 70 72.5 


3.4. Experiment-4 

In this experiment, the second designing scenario shown in Figure 3(b) was applied to the display 
again. The experimental protocol also includes four trials in which the subject was asked to follow a specific 
path (directions) on the ground using a motorized wheelchair (its speed | m/s). The display with scenario 2 is 
attached to this wheelchair. The path shown in Figure 4 is the same path that subjects should follow. The 
length of the path is 21 meters, in which the subject expected to select button B=zero times, F=six times, 
S=five times, R=three times, L=twice and double blink (D.B)=once only to complete the path. The ideal 
consumed time (when no selection mistakes are committed) for this scenario (horizontal bar) equal to 84 
seconds. Subjects may commit to types of mistakes when using this system; missing specific selection and 
select wrong direction. Every missed selection will add five seconds (1-cycle scanning time) to the total 
consumed time. Whilst selecting a wrong button (different direction) will require additional time for multi 
selections until the direction is corrected. The period is theoretically calculated according to (2) which is 
based on the system scanning and selecting capabilities. Table 5 shows the time consumed by each subject to 
complete the path in four trials. 


Consumed Time = 2B+3L+S+5R+6F+D.B+ X (2) 


Where: 

B=the no. of backward button selections 

L=the no. of left button selections 

S=the no. of stop button selections 

R=the no. of right button selections 

F=the no. of forward button selections 

D.B=the no. of double blinking (emergency stop) 
X=the length of the path 


Table 5. The subject consumed timed per trial to complete the path using eye-blinking in the 2™ scenario 
(Trials) consumed time in sec /path 


Subject No. Trial 1 Trial 2 Trial 3 Trial 4 Average consumed time (sec) 
Sl 110 105, 115 110 110 
S2 105, 105, 100 95 101.25 
S3 110 100 100 95 101.25 
S4 105 100 105 100 102.5 
S5 110 100 105 100 103.75 


4. DISCUSSION 

According to Table 2, results show that the average number of button selection falls between 3.8 and 
4.6 while the mean standard deviation equals to 0.646. Trial 3 (for left button) shows the highest average 
number of selections (4.6 times/30 sec) whilst trail 2 (backward button) and trial 4 (right button) show the 
lowest average number (3.8 times/30 sec). Furthermore, the maximum recorded no. of selection in all trials is 
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5 times/30 sec and the minimum one is 3 times/30 sec. On the other hand, Table 3 shows that that the average 
number of button selection falls between 3.6 and 4.2 while the mean standard deviation equals to 0.415. Trial 
2 (backward button) shows the highest average number of selections (4.2 times/30 sec) whilst trial 4 (right 
button) show the lowest average number (3.6 times/30 sec). The mean standard deviation of second 
designing scenario conducted in experiment 2 is lower than the first scenario (done in experiment 1), thus, 
the second scenario considered more reliable than the first one in the process of button selecting through a 
scanning session. 

In validating the system through navigation, Table 4 shows that the subjects average consumed time 
was between (72.5-86.25) sec to complete the set path using the first scenario (cross shape) which is 
approximately close to the theoretical calculation (60 sec/path). Whilst, Table 5 shows a dramatic increase in 
the consumed time for subjects using the second scenario (horizontal bar) which falls between 
(101.25—110) sec to complete the set path which is relatively high in comparison to the theoretical calculation 
(84 sec/path). Consequently, using the first scenario (cross-shaped) in navigating the wheelchair consuming 
less time than using the second scenario (horizontal bar) for the same path. Therefore, the first scenario is 
recommended for navigation but it needs training sessions for subjects to use it easily. 


5. CONCLUSION 

This paper proposed a human-machine interaction for motorized wheelchair based on single- 
channel EEG headband. The system develops a simple HMI to assist fully/partially paralyzed people to move 
freely in their daily life using a motorized wheelchair. The proposed system was successfully validated 
through sets of practical experiments and two designed scenarios were applied. Five normal subjects were 
involved in the validation process and four trials were accomplished for each experiment. Four experiments 
were done to validate the designed scenarios by calculating the mean standard deviation and the average 
consumed time for each one of them. Choosing the proper scenario was based on the user’s awareness 
capability in selecting on screen buttons and the system scanning time. The horizontal bar scenario 
considered the easier one in buttons selection process whilst the cross-shaped was the best one in the 
navigation. The system is cost effective and easy to use; however, a user training session is essential. 


6. FUTURE WORK 

The implemented system can be equipped with a global positioning system (GPS) technology. 
Adding this component has several benefits to the developed HMI system. The first benefit is the patient 
safety which means the wheelchair can be easily monitored remotely while another one is the navigation. In 
this case, patients can create their own path for movement. Sensors such as ultrasound and accelerometer 
could also be added to the system in order to increase the reliability. These components collect data and let 
patients to interact with their environment. 
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